[fix] [broker] Part-2: Replicator can not created successfully due to an orphan replicator in the previous topic owner #21948

poorbarcode · 2024-01-22T21:23:53Z

Motivation

There is a race condition that makes an orphan replicator in the original owner of a topic, and causes the new owner of the topic can not start a replicator due to org.apache.pulsar.broker.service.BrokerServiceException$NamingException Producer with name 'pulsar.repl.{local_cluster}-->{remote_cluster}' is already connected to topic.

Scenario 1

Thread-1: start/restart the producer of the replicator.
Thread-2: unloading bundles.

Scenario 2

Thread-1: start a new replicator after updated replication_clusters.
Thread-2: unloading bundles.

After we solved the scenario 1 by #21946, the current PR is focusing on the scenario 2:

Current PR is focusing on Scenario 2.

Steps of Scenario 2

time	`thread enable replication`	thread `unload bundle`
1	Enabled replication
2		Mark topic as `closing`
3		Skip `replicator.disconnect()` because `topic.replicators` is empty
4	Initialize cursor `pulsar.repl`
5	Start producer
6	Set `replicator.stat --> Starting`
7	Create producer success and set `replicator.stat --> Started`
8	Trigger a `readMoreEntries`, since there is no entries to read, just pending this request
9		Close cursor `pulsar.repl`
10		Close managed ledger
11	An orphan replicator is there, and the next topic owner could not start a replicator due to `Producer with name 'pulsar.repl.{local_cluster}-->{remote_cluster}' is already connected to topic`

Since the scenario is too complex, I can not add a test.

TODO: reproduce the Scenario 2 locally.

Modifications

call replicators.disconnect after the managed ledger is closed. It would prevent the new cursor(pulsar.dedup) from being created.
topic.close will be done after replicators.disconnect, it can avoid the new replicator on the next owner broker of the topic failing due to creating an internal producer failed org.apache.pulsar.broker.service.BrokerServiceException$NamingException Producer with name 'pulsar.repl.{local_cluster}-->{remote_cluster}' is already connected to topic.
After [fix] [broker] Replication stopped due to unload topic failed #21947 the operation replicator.producer.close will no longer fail.

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository: x

codelipenghui

It looks like we can also fix the issue by involving a read lock to the addReplicationCluster() method in PersistentTopic.java without changing the close topic process. The replicator.disconnect() method will also call the managed cursor API to change the cursor state which doesn't make sense to close the managed ledger before closing the replicator

diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index 9baafcb2e9..db6910eb51 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -1918,11 +1918,16 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
                         return;
                     }
                     Replicator replicator = replicators.computeIfAbsent(remoteCluster, r -> {
+                        lock.readLock().lock();
                         try {
-                            return new GeoPersistentReplicator(PersistentTopic.this, cursor, localCluster,
-                                    remoteCluster, brokerService, (PulsarClientImpl) replicationClient);
+                            if (!isClosingOrDeleting) {
+                                return new GeoPersistentReplicator(PersistentTopic.this, cursor, localCluster,
+                                        remoteCluster, brokerService, (PulsarClientImpl) replicationClient);
+                            }
                         } catch (PulsarServerException e) {
                             log.error("[{}] Replicator startup failed {}", topic, remoteCluster, e);
+                        } finally {
+                            lock.readLock().unlock();
                         }
                         return null;
                     });

… an orphan replicator in the previous topic owner

poorbarcode · 2024-04-24T07:28:17Z

Rebase master

poorbarcode · 2024-04-24T10:09:29Z

@codelipenghui

It looks like we can also fix the issue by involving a read lock to the addReplicationCluster() method in PersistentTopic.java without changing the close topic process. The replicator.disconnect() method will also call the managed cursor API to change the cursor state which doesn't make sense to close the managed ledger before closing the replicator

Good suggestion, already improved the code.

codecov-commenter · 2024-04-24T17:39:24Z

Codecov Report

Attention: Patch coverage is 67.56757% with 12 lines in your changes are missing coverage. Please review.

Project coverage is 73.95%. Comparing base (bbc6224) to head (3d389b1).
Report is 186 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #21948      +/-   ##
============================================
+ Coverage     73.57%   73.95%   +0.38%     
+ Complexity    32624     2744   -29880     
============================================
  Files          1877     1885       +8     
  Lines        139502   140512    +1010     
  Branches      15299    15450     +151     
============================================
+ Hits         102638   103916    +1278     
+ Misses        28908    28563     -345     
- Partials       7956     8033      +77

Flag	Coverage Δ
inttests	`26.98% <21.62%> (+2.39%)`	⬆️
systests	`24.47% <10.81%> (+0.15%)`	⬆️
unittests	`73.24% <67.56%> (+0.39%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...a/org/apache/pulsar/broker/service/Replicator.java	`0.00% <ø> (ø)`
...roker/service/persistent/PersistentReplicator.java	`66.21% <75.00%> (-2.66%)`	⬇️
...ache/pulsar/broker/service/AbstractReplicator.java	`62.06% <50.00%> (-22.94%)`	⬇️
...sar/broker/service/persistent/PersistentTopic.java	`78.46% <68.96%> (+<0.01%)`	⬆️

... and 262 files with indirect coverage changes

lhotari · 2024-04-25T14:47:40Z

@poorbarcode I wonder if these changes somehow cause OOME problems with org.apache.pulsar.broker.service.ReplicatorSubscriptionTest#testWriteMarkerTaskOfReplicateSubscriptions, please see
#22586 . Thanks

lhotari · 2024-04-25T22:18:01Z

@poorbarcode I can confirm that this PR is causing problems which most likely lead to OOMEs. Reported as #22588.
To reproduce:

git clone https://github.com/apache/pulsar
cd pulsar
# checkout the merge commit for #21948
git checkout b774666
mvn -Pcore-modules,-main -T 1C clean install -DskipTests -Dspotbugs.skip=true -Dcheckstyle.skip=true
mvn -DredirectTestOutputToFile=false -DtestRetryCount=0 test -pl pulsar-broker "-Dtest=org.apache.pulsar.broker.service.ReplicatorGlobalNSTest#testRemoveLocalClusterOnGlobalNamespace" | tee test_$(git rev-parse --short HEAD).log

then checkout d475655 which is the commit before b774666 (#21948). you can see that the behavior is very different:

git checkout d475655
mvn -Pcore-modules,-main -T 1C clean install -DskipTests -Dspotbugs.skip=true -Dcheckstyle.skip=true
mvn -DredirectTestOutputToFile=false -DtestRetryCount=0 test -pl pulsar-broker "-Dtest=org.apache.pulsar.broker.service.ReplicatorGlobalNSTest#testRemoveLocalClusterOnGlobalNamespace" | tee test_$(git rev-parse --short HEAD).log

heesung-sn · 2024-04-25T22:53:06Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java

+    }
+
+    private void unfenceReplicatorsToResume() {
+        checkReplication();


I think this can lead to a loop
checkReplication -> fail to delete replicated topic -> unfenceTopicToResume ->checkReplication

2024-04-25T14:57:46,573 - INFO - [broker-topic-workers-OrderedExecutor-2-0:PersistentTopic] - Deleting topic [persistent://pulsar/global/removeClusterTest/__change_events] because local cluster is not part of global namespace repl list [r2, r3] 2024-04-25T14:57:46,573 - INFO - [ForkJoinPool.commonPool-worker-4:AbstractReplicator] - [persistent://pulsar/global/removeClusterTest/__change_events | r1-->r2] Skip current termination since other thread is doing termination, state : Terminated 2024-04-25T14:57:46,573 - INFO - [ForkJoinPool.commonPool-worker-4:AbstractReplicator] - [persistent://pulsar/global/removeClusterTest/__change_events | r1-->r3] Skip current termination since other thread is doing termination, state : Terminated 2024-04-25T14:57:46,573 - ERROR - [broker-topic-workers-OrderedExecutor-2-0:PersistentTopic] - [persistent://pulsar/global/removeClusterTest/__change_events] Error deleting topic org.apache.pulsar.broker.service.BrokerServiceException$PersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: org.apache.bookkeeper.mledger.ManagedLedgerException$CursorAlreadyClosedException: Cursor was already closed at org.apache.pulsar.broker.service.persistent.PersistentTopic$6.deleteLedgerFailed(PersistentTopic.java:1496) ~[classes/:?] at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.lambda$asyncDelete$33(ManagedLedgerImpl.java:2950) ~[managed-ledger-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2357) ~[?:?] at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.asyncDelete(ManagedLedgerImpl.java:2947) ~[managed-ledger-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT] at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$delete$40(PersistentTopic.java:1468) ~[classes/:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2357) ~[?:?] at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$delete$41(PersistentTopic.java:1462) ~[classes/:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2357) ~[?:?] at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$delete$42(PersistentTopic.java:1453) ~[classes/:?] at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$delete$34(PersistentTopic.java:1431) ~[classes/:?] at java.base/java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:787) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) ~[?:?] at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[bookkeeper-common-4.17.0.jar:4.17.0] at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:107) ~[bookkeeper-common-4.17.0.jar:4.17.0] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.108.Final.jar:4.1.108.Final] at java.base/java.lang.Thread.run(Thread.java:1570) [?:?] Caused by: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: org.apache.bookkeeper.mledger.ManagedLedgerException$CursorAlreadyClosedException: Cursor was already closed Caused by: java.util.concurrent.CompletionException: org.apache.bookkeeper.mledger.ManagedLedgerException$CursorAlreadyClosedException: Cursor was already closed at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.andTree(CompletableFuture.java:1527) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.allOf(CompletableFuture.java:2451) ~[?:?] at org.apache.pulsar.common.util.FutureUtil.waitForAll(FutureUtil.java:56) ~[pulsar-common-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT] at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.asyncTruncate(ManagedLedgerImpl.java:4341) ~[managed-ledger-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT] at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.asyncDelete(ManagedLedgerImpl.java:2946) ~[managed-ledger-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT] ... 19 more Caused by: org.apache.bookkeeper.mledger.ManagedLedgerException$CursorAlreadyClosedException: Cursor was already closed 2024-04-25T14:57:46,573 - INFO - [broker-topic-workers-OrderedExecutor-2-0:PersistentTopic] - Deleting topic [persistent://pulsar/global/removeClusterTest/__change_events] because local cluster is not part of global namespace repl list [r2, r3] 2024-04-25T14:57:46,573 - INFO - [ForkJoinPool.commonPool-worker-4:AbstractReplicator] - [persistent://pulsar/global/removeClusterTest/__change_events | r1-->r2] Skip current termination since other thread is doing termination, state : Terminated 2024-04-25T14:57:46,573 - INFO - [broker-topic-workers-OrderedExecutor-2-0:BrokerService] - Successfully delete authentication policies for topic persistent://pulsar/global/removeClusterTest/__change_events 2024-04-25T14:57:46,573 - INFO - [ForkJoinPool.commonPool-worker-4:AbstractReplicator] - [persistent://pulsar/global/removeClusterTest/__change_events | r1-->r3] Skip current termination since other thread is doing termination, state : Terminated 2024-04-25T14:57:46,573 - INFO - [broker-topic-workers-OrderedExecutor-2-0:ManagedLedgerImpl] - pulsar/global/removeClusterTest/persistent/__change_events Moving to FencedForDeletion state 2024-04-25T14:57:46,573 - ERROR - [broker-topic-workers-OrderedExecutor-2-0:ManagedLedgerImpl] - [pulsar/global/removeClusterTest/persistent/__change_events] Error truncating ledger for deletion java.util.concurrent.CompletionException: org.apache.bookkeeper.mledger.ManagedLedgerException$CursorAlreadyClosedException: Cursor was already closed at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.andTree(CompletableFuture.java:1527) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.allOf(CompletableFuture.java:2451) ~[?:?] at org.apache.pulsar.common.util.FutureUtil.waitForAll(FutureUtil.java:56) ~[pulsar-common-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT] at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.asyncTruncate(ManagedLedgerImpl.java:4341) ~[managed-ledger-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT] at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.asyncDelete(ManagedLedgerImpl.java:2946) ~[managed-ledger-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT] at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$delete$40(PersistentTopic.java:1468) ~[classes/:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2357) ~[?:?] at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$delete$41(PersistentTopic.java:1462) ~[classes/:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2357) ~[?:?] at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$delete$42(PersistentTopic.java:1453) ~[classes/:?] at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$delete$34(PersistentTopic.java:1431) ~[classes/:?] at java.base/java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:787) ~[?:?] at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) ~[?:?] at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[bookkeeper-common-4.17.0.jar:4.17.0] at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:107) ~[bookkeeper-common-4.17.0.jar:4.17.0] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.108.Final.jar:4.1.108.Final] at java.base/java.lang.Thread.run(Thread.java:1570) [?:?] Caused by: org.apache.bookkeeper.mledger.ManagedLedgerException$CursorAlreadyClosedException: Cursor was already closed

I think this also means we need to think about how to skip deleting ledgers when the cursor is closed.

It is caused by the method topic.close is a reentrant method, the issue steps are like the following

reset namespace level policies -> checkReplication -> delete topic

unload namespace -> unload topic -> close topic

The method topic.close can be run even if the deleting task is in-progress, it is not correct, the PR #17524 will fix it, please review it again, thanks

Can we define enum state for topic and reject/skip any invalid state transition (fail/skip fast)?

For example, delete should skip if closed, and vice-versa.

Agree with you, we need more than one separate PRs to merge the states fenced, close, deleting, just like we discussed here

… an orphan replicator in the previous topic owner (#21948) (cherry picked from commit b774666)

… an orphan replicator in the previous topic owner (apache#21948) (cherry picked from commit b774666) (cherry picked from commit 6038bbf)

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jan 22, 2024

poorbarcode mentioned this pull request Jan 22, 2024

[fix] [broker] Part-1: Replicator can not created successfully due to an orphan replicator in the previous topic owner #21946

Merged

4 tasks

poorbarcode self-assigned this Jan 24, 2024

poorbarcode added type/bug The PR fixed a bug or issue reported a bug release/2.10.6 category/reliability The function does not work properly in certain specific environments or failures. e.g. data lost release/3.0.3 release/2.11.4 release/3.1.3 labels Jan 24, 2024

poorbarcode added this to the 3.3.0 milestone Jan 24, 2024

Technoboy- added the ready-to-test label Jan 25, 2024

Technoboy- closed this Jan 25, 2024

Technoboy- reopened this Jan 25, 2024

Technoboy- approved these changes Jan 25, 2024

View reviewed changes

codelipenghui reviewed Jan 29, 2024

View reviewed changes

poorbarcode force-pushed the fix/orphan_replicator_4 branch from 8d24801 to 0723666 Compare January 29, 2024 15:37

poorbarcode force-pushed the fix/orphan_replicator_4 branch from 0723666 to c163106 Compare April 1, 2024 14:49

poorbarcode force-pushed the fix/orphan_replicator_4 branch from c163106 to 1506b48 Compare April 24, 2024 07:24

poorbarcode added 3 commits April 24, 2024 15:25

[fix] [broker] Part-2: Replicator can not created successfully due to…

8ee62c7

… an orphan replicator in the previous topic owner

make up the missing part in Part-1

c216c09

fix test

d3b00d2

poorbarcode force-pushed the fix/orphan_replicator_4 branch from 1506b48 to d3b00d2 Compare April 24, 2024 07:28

poorbarcode added 2 commits April 24, 2024 16:26

use new solution

0ee09e1

add test: testUnFenceTopicToReuse

211f8b5

poorbarcode requested a review from codelipenghui April 24, 2024 10:09

codelipenghui approved these changes Apr 24, 2024

View reviewed changes

poorbarcode added 2 commits April 24, 2024 19:30

fix checkstyle

5735988

fix tests

3d389b1

poorbarcode merged commit b774666 into apache:master Apr 24, 2024
50 checks passed

lhotari mentioned this pull request Apr 25, 2024

[improve][ci] Disable test that causes OOME until the problem has been resolved #22586

Merged

4 tasks

lhotari mentioned this pull request Apr 25, 2024

[Tests][Bug] There are multiple memory leaks that cause Pulsar CI to fail with OOME #22588

Closed

3 tasks

heesung-sn reviewed Apr 25, 2024

View reviewed changes

lhotari mentioned this pull request Apr 26, 2024

[fix][broker][WIP] Don't check replication clusters for ownership of system topics #22597

Draft

4 tasks

poorbarcode added a commit that referenced this pull request May 7, 2024

[fix] [broker] Part-2: Replicator can not created successfully due to…

788124d

… an orphan replicator in the previous topic owner (#21948) (cherry picked from commit b774666)

poorbarcode added cherry-picked/branch-3.2 release/3.2.3 and removed release/3.1.3 labels May 7, 2024

poorbarcode added a commit that referenced this pull request May 7, 2024

[fix] [broker] Part-2: Replicator can not created successfully due to…

6038bbf

… an orphan replicator in the previous topic owner (#21948) (cherry picked from commit b774666)

poorbarcode added cherry-picked/branch-3.0 and removed release/2.10.6 labels May 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix] [broker] Part-2: Replicator can not created successfully due to an orphan replicator in the previous topic owner #21948

[fix] [broker] Part-2: Replicator can not created successfully due to an orphan replicator in the previous topic owner #21948

poorbarcode commented Jan 22, 2024 •

edited

Loading

codelipenghui left a comment

poorbarcode commented Apr 24, 2024

poorbarcode commented Apr 24, 2024

codecov-commenter commented Apr 24, 2024

lhotari commented Apr 25, 2024 •

edited

Loading

lhotari commented Apr 25, 2024 •

edited

Loading

heesung-sn Apr 25, 2024 •

edited

Loading

heesung-sn Apr 25, 2024

heesung-sn Apr 25, 2024 •

edited

Loading

poorbarcode Apr 26, 2024

heesung-sn Apr 26, 2024

poorbarcode Apr 26, 2024 •

edited

Loading

[fix] [broker] Part-2: Replicator can not created successfully due to an orphan replicator in the previous topic owner #21948

[fix] [broker] Part-2: Replicator can not created successfully due to an orphan replicator in the previous topic owner #21948

Conversation

poorbarcode commented Jan 22, 2024 • edited Loading

Motivation

Modifications

Documentation

Matching PR in forked repository

codelipenghui left a comment

Choose a reason for hiding this comment

poorbarcode commented Apr 24, 2024

poorbarcode commented Apr 24, 2024

codecov-commenter commented Apr 24, 2024

Codecov Report

lhotari commented Apr 25, 2024 • edited Loading

lhotari commented Apr 25, 2024 • edited Loading

heesung-sn Apr 25, 2024 • edited Loading

Choose a reason for hiding this comment

heesung-sn Apr 25, 2024

Choose a reason for hiding this comment

heesung-sn Apr 25, 2024 • edited Loading

Choose a reason for hiding this comment

poorbarcode Apr 26, 2024

Choose a reason for hiding this comment

heesung-sn Apr 26, 2024

Choose a reason for hiding this comment

poorbarcode Apr 26, 2024 • edited Loading

Choose a reason for hiding this comment

poorbarcode commented Jan 22, 2024 •

edited

Loading

lhotari commented Apr 25, 2024 •

edited

Loading

lhotari commented Apr 25, 2024 •

edited

Loading

heesung-sn Apr 25, 2024 •

edited

Loading

heesung-sn Apr 25, 2024 •

edited

Loading

poorbarcode Apr 26, 2024 •

edited

Loading